Local adaptive algorithms for information maximization in neural networks, and application to source separation

Information theoretic criteria for neural network adaptation laws have recently become an important focus of attention. We consider the problem of adaptively maximizing the entropy of the outputs of a deterministic feedforward neural network with real valued stochastic input signals, as considered by Bell and Sejnowski. We give a new explanation for the relevance of output information (entropy) maximization for source separation applications and reinterpret Bell and Sejnowski's approach in a more general context of probability density estimation. This insight is the basis for a generalization of the approach, and we consider a family of gradient based algorithms. 1. BLIND SOURCE SEPARATION, INFORMATION MAXIMIZATION AND PROBABILITY ESTIMATION The problem of blind separation of independent sources can be formulated as follows. A vector of n stationary input signals x(t) 2 R is known to result from mixing n stochastically independent sources s(t) 2 R :